Basic Troubleshooting Labs

All labs in this module will run on server1.example.com.

Goals

Find logs with recent entries
Find relevant data in log files, commands, system files, and configuration files
Increase verbosity in system commands
Find differences in files
Run daemons in debug mode
Run daemons with increased log verbosity

1. Gathering Relevant Data from the System

Check which log files were updated recently.
```
# cd /var/log
# ls -ltr
```
Look at each of the following items.
- Boot time kernel messages:
  # less /var/log/dmesg
- The latest kernel messages:
  # dmesg|tail
- Any entries in /var/log/messages pertaining to eth0 but exclude any entries containing DHCP:
  # grep -i eth0 /var/log/messages*|grep -v DHCP
- The rsyslogd process in ps output:
  # ps ax |grep rsyslogd
- The rsyslogd or init processes in ps output:
  # ps ax | grep -E "rsyslogd|init"
- The first line in /etc/passwd:
  # head -1 /etc/passwd
- The last two lines in /etc/passwd:
  # tail -2 /etc/passwd
Show the output of a command, then show it again but remove the first line (header text) only.
```
# ps
# ps | tail -n +2
```
Get the output of df (-P removes line breaks), remove the header text, and only print the last column which consists of the mounted file systems.
```
# df -P | tail -n +2 |awk '{print $6}'
```
Run the lspci command with extra verbosity.
```
# lspci -vvv|less
```
Install the screen package.
1. Create a screen RC file called /root/monitor.scr with the following content:
  screen -t "Log Monitor" 1 tail -f /var/log/messages screen -t "Command Window" 2 bash split select 1 focus select 2 startup_message off
2. Start the split screen log monitor.
  # screen -c /root/monitor.scr
3. Send an update to /var/log/messages.
  # logger "This is a test"
  - To scroll back on the top window use CTRL+a, release, then press TAB.
  - To enter copy mode, use CTRL+a, release, then press [.
    When in copy mode, use j for up and k for down or use the up/down arrow keys.
  - Use CTRL+a, then press TAB to toggle back to the command line screen.
  - To exit the screen, use CTRL+a, release, then type :quit and press Enter.

2. Comparing Files

Create a copy of the /etc/ssh/sshd_config file, make a change to it and see how diff can find differences between files:

# cp /etc/ssh/sshd_config /root/mysshd_config
# vim /root/mysshd_config
<uncomment and change LogLevel to DEBUG>
LogLevel DEBUG
<uncomment and change Port to 2222>
Port 2222

# diff /etc/ssh/sshd_config /root/mysshd_config

3. Running Services Verbosely

Create a screen RC file called /root/monitor_sshd.scr.

This starts a test instance of sshd when you start screen.

screen -t "SSHD Monitor"   1 /usr/sbin/sshd -De -f /root/mysshd_config
screen -t "Command Window" 2 bash
split
select 1
focus
select 2
startup_message off

Start the split screen sshd monitor.
```
# screen -c /root/monitor_sshd.scr
```
Use ssh to connect to the test sshd instance from the bottom screen.
```
# ssh localhost -p 2222
```

4. Basic Service Troubleshooting

Make sure Apache is installed and started.

# yum -y install httpd
# systemctl start httpd
# systemctl enable httpd

Create a default index page.

# echo "This is the default site" > /var/www/html/index.html

Test the http service locally.

# curl http://localhost
This is the default site

Create a new directory and content for Apache.

# mkdir /var/www/mysite
# echo "This is my site" > /var/www/mysite/index.html

Create a new configuration file in Apache in a file called /etc/httpd/conf.d/my.conf for the new content directory (use this text EXACTLY):

# vim /etc/httpd/conf.d/my.conf
Alias /mysite "/var/www/my_site"

<Directory "/var/www/mysite">
    Options Indexes MultiViews FollowSymLinks
    AllowOverride None
    Order allow,deny
    Require all granted
</Directory>

In a separate terminal (use screen if you like), monitor the Apache error log.
```
# tail -f /var/log/httpd/error_log
```
Restart Apache, then try to access the new content (it should fail).
```
# systemctl reload httpd
# curl http://localhost/mysite/
```
Using the information gained from monitoring error_log, fix my.conf, and then try to access the site again.
```
# vim /etc/httpd/conf.d/my.conf
<fix error>
# systemctl reload httpd
# curl http://localhost/mysite/
This is my site
```
Be sure to specify the trailing / in the curl command. Repeat until it’s fixed.